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ABSTRACT 



This paper summarizes results of telephone interviews with 
key personnel at 19 (out of 22) agencies receiving grants under the Center 
for Mental Health Services program, one part of a 5 -year evaluation study 
examining the impact that utilizing measurable outcomes has on service 
systems. This component focused on the conceptualization and implementation 
of outcome-based information systems and involved interviews with sites which 
had and had not established an outcome information system. Results are 
reported across three domains: stakeholder involvement, impact on service 
planning and delivery, and likelihood of continuing outcome monitoring after 
completion of the national evaluation. Results suggest that for outcomes to 
have a maximum positive impact on service planning and delivery, there needs 
to be a high degree of overlap between the service system and the information 



system. (DB) 
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The Building Outcome Accountability in Children's Mental 
Health project was an effort to learn more about how each 
of the twenty-two Center for Mental Health Services 
(CMHS) grantees were conceptualizing and implementing a 
local process of outcome evaluation. This study was 
conducted as part of the System Accountability Study, a 
5-year research effort of the Research and Training Center 
for Children's Mental Health (RTC) designed to investigate 
the impact that utilizing measurable outcomes has on 
service systems. 



The Building Outcome Accountability project focused on 
the conceptualization and implementation of outcome-based 
information systems at the twenty-two CMHS sites. These 
sites received grants designed to promote the development 
of systems of care that include mental health, child welfare, 
education, juvenile justice, and other appropriate agencies to 
meet the multiple and changing needs of children and 
adolescents and their families. A related activity, the core 
evaluation of the CMHS initiative, conducted by Macro 
International, Inc. and its partner, the University of South 
Florida, is focused on the child, the system, and the 
interaction between the two. The Building Outcome 
Accountability project focused on how outcome results 
were being used in CMHS systems of care in a effort to 
understand how using measurable outcome data affects 
service planning and delivery. 






Methodology 

The data for the Building Outcome Accountability project 
were collected through a series of structured key informant 
telephone interviews with employees at each of the CMHS 
sites. Two interview protocols were developed for this 
research project: 1) Survey A, designed for sites with 
established outcome information systems; and 2) Survey B, 
designed for sites which had not yet established outcome 
information system. Each site determined who would 
participate in the interview and whether Survey A or Survey 
B would be completed. 

Both survey instruments asked participants to comment on 
the array of services offered by their system of care, how 
and when the system of care was established, and 
demographic information about the population served. Both 
surveys also asked participants to comment on the 
involvement of stakeholders in the process of selecting 
outcomes to be tracked and the impact of outcome tracking 
on service planning and delivery. In addition, systems which 
identified themselves as having established outcome 
tracking systems were asked to comment on what kinds of 
outcomes were tracked, how often they are tracked, how 
these outcomes are measured, and the level of interagency 
collaboration required to monitor outcomes. They were also 
asked to discuss how outcome information is being 
reported, who receives these reports, and how it is being 
used. Finally, respondents that completed the survey 
questions for sites with an established outcome information 
system were asked to comment on the continued use of 
outcome data when the national evaluation is completed. 

The interviews for this project were conducted during a 
five-month period beginning October 1995 and ending in 
February 1996. Nineteen out of the twenty -two grantee sites 
(86%) participated in the interview process. All participants 
were advised that the results of this study would be reported 
in aggregated form and that the identities of individual 
systems of care and respondents would remain anonymous. 
Ten of the respondents (53%) represented sites with an 
established outcome information system, while nine 
participating sites (47%) did not have such a system 
established. 

Results 

The data was analyzed across three domains; 1) stakeholder 
involvement in selecting outcomes; 2) the impact of 
outcomes on service planning and delivery; and 3) using 
outcomes when the national evaluation is completed. The 
results are discussed below. 

Stakeholder Involvement 

Participants were asked to rate the importance of 
stakeholder involvement on a Likert-type scale from 1 - not 
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stakeholder involvement on a Likert-type scale from 1- not 
important at all to 5- crucial. All but one of the 
participating sites considered stakeholder involvement in the 
process of selecting outcomes to be very important or 
higher. The single site not responding in this range reported 
that they chose not to respond, believing it to be irrelevant 
to their situation because they had uniformly adopted the 
outcome measures currently being used in the national 
evaluation. While nineteen sites considered stakeholder 
involvement important to the process of selecting outcomes, 
5 sites (one site which considered stakeholder involvement 
very important and four which considered it crucial ) 
reported having no vehicle currently developed to facilitate 
this involvement. One respondent commented that there is 
no systematic way of involving stakeholders at this time, 
although they anticipated doing so in the future. Another 
commented that there is a "growing discussion of how 
stakeholders can be involved." These results imply that 
while almost all of the respondents thought stakeholder 
involvement was important, some sites (26%) had no 
mechanism for achieving this goal. This indicates that 
agreement with the principle does not always equate to 
implementation. 

Impact on Service Planning and Delivery 

Nineteen of the twenty participants in this study chose to 
answer the series of questions relating to the impact of 
outcome tracking on service planning and delivery. One site 
declined because they believed there was not yet enough 
outcome data available yet to comment. Respondents were 
asked to rank the impact of outcome 
tracking&endash;anticipated or actual&endash;on service 
planning and service delivery on a Likert-type scale of 1- «o 
impact to 5 - tremendous impact. Ten participants (53%) 
responded that the impact on services planning was 
tremendous, eight responded the. impact was 4 - 
considerable, and one responded outcome tracking had 3 - 
some impact. The ten sites which rated the impact of 
outcomes on service planning as tremendous also believed 
the impact of outcome tracking on service delivery to be 
tremendous. Of the remaining nine sites, six rated the 
impact of outcomes on service delivery as considerable, and 
three believed outcomes would have some impact on service 
planning. 

A common concern expressed by respondents was for the 
burden and consequent stress that the process of establishing 
a system of outcome information places on management and 
staff. One respondent commented, "Putting an outcome 
system in place can be painful because of the procedural 
changes, increases in paperwork, and increased need for 
management and staff training." Other respondents echoed 
this statement. Concern was expressed that, "outcome 
tracking adds paperwork and takes time away from actual 
service provision." Another commented that "tracking 
outcomes requires ongoing training and technical assistance 
on how to and why outcomes are collected. People are 
intimidated by data and forms. You must keep reinforcing 
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intimidated by data and forms. You must keep reinforcing 
the process." 

The experience of staff at CMHS sites which have 
established outcome information systems indicates that the 
stress and burden of implementation may be a short-term 
concern. One respondent discussed the short term impact: 
"Service providers are heavily impacted by the demands of 
the outcome process. It is difficult to implement in the 
short-run because the most immediate impact is that 
clinicians are overwhelmed by the process." According to 
some respondents, however, once outcome information 
becomes available, stress-levels are reduced. 

Several respondents expressed concern for fears associated 
with how outcome information is used. One respondent 
stated, "there exists a real fear on the part of clinicians that 
outcomes will be used to demonstrate they are not doing a 
good job &endash; what if we're not doing well?" There 
was consensus that training was an important way to 
promote understanding of how outcomes can be used in a 
positive way to improve service delivery. 

Continuation of Outcome Monitoring 

Respondents representing the ten sites with established 
outcome information systems were asked a series of 
questions about whether, and under what circumstances, 
they might continue tracking outcome information after 
completion of the national evaluation. All ten responded 
that they anticipate continuing the process of outcome 
monitoring, although the strength of the replies ranged from 
"Yes, hopefully, it depends on available funding" to "Yes, 
definitely, outcomes very similar to the national evaluation 
are included in our state plan." When asked to identify 
factors that would influence the decision to continue 
outcome tracking, it is noteworthy that eight of the ten sites 
mentioned available funding as the central deciding factor. 

Conclusions 

It was recognized early in the development of the interview 
instruments used in this project that there would be variation 
among the CMHS grantees in the degree to which their 
systems of care had been conceptualized and implemented, 
as well as the degree to which their information 
accountability systems were developed. The amount of 
overlap between the level of development of the service 
system and the outcome information system seemed to have 
an impact on how outcome information is used by the 
system of care (see Figure 1) . This can be seen in how 
information about clinical outcome measures is used. 
Descriptions of three example sites follow to illustrate 
varying degrees of overlap between the level of 
development of the service systems and the accountability 
systems in this study. 

A respondent whose system of care began developing 
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A respondent whose system of care began developing 
within the past few years reports that the site is also just 
beginning to collect outcome information. This site 
respondent indicates that the system does not have a vehicle 
for involving stakeholders in decisions regarding outcomes 
at this time. Interagency collaboration at the site occurs only 
at the direct-service level where interagency teams serve 
individual children and their families. Although this 
respondent noted that the "CAFAS [Child and Adolescent 
Functional Assessment Scale] is an excellent measure," staff 
are still struggling with trying to determine what outcomes 
they need to track and how to use the results. This system 
seems to represent a situation in which the service system 
and the outcome accountability system are both developing 
in parallel and exist independently of one another at this 
point in time. 

Another grantee in the early stages of system development 
describes their system as having "lots of pieces for the past 
eight or so years, but we only recently began building an 
integrated system." This respondent describes the site as 
relying heavily on the input of a broad range of stakeholders 
as it expands its array of services. Also, interagency 
relationships are growing in strength and he/she believes 
that "to take a systems point of view, we need to know what 
our connection with other systems is." This respondent 
reports that the site is already using clinical data to make 
decisions at the treatment level, but does not yet have a way 
of using this information at a program or system level. This 
site represents a system at the early stages of both system 
and information development. Some degree of overlap in 
the development of the services offered and the information 
system seems to be indicated by the use of clinical outcome 
data to inform treatment at the child and family level. 

In contrast to the previous examples, one of the more 
developed systems of care sites has already focused on 
potential uses for outcome results or information at the 
system level. The respondent for this site reported that its 
system of care had been developed for more than ten years, 
and it had been measuring and reporting outcome 
information for several years. The initial outcomes tracked 
through the system of care were more global; for example, 
emphasis was placed on system level measures such as 
out-of home placements, school attendance and 
achievement, and juvenile justice recidivism rather than 
clinical measures. Discussions at the site concerning 
different aspects of the information system focused on 
issues of sharing information across the system, such as 
creating electronically connected information systems to 
lower technical boundaries so that outcome data could be 
accessed. This site's focus on clinical outcome measures 
went beyond their use at a child level/direct service level to 
recognition of how clinical status information might be used 
to learn more about how effectively the system is 
functioning. This site seems to represent a system of care 
which is well developed with an information system which 
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is well established. In addition, there seems to be a high 
degree of overlap between the system of services and the 
information system. This takes place through its efforts to 
create a seamless source of interagency information and its 
interest in using clinical outcome information to evaluate 
system effectiveness in addition to informing treatment 
decisions. 

In conclusion, the CMHS sites represent varying degrees of 
both systems development outcome accountability 
development. For outcomes to have a maximum positive 
impact on service planning and delivery, it may be 
necessary for there to be a high degree of overlap between 
the service system and the information system. The data 
collected in this study illustrate a range of overlap from a 
service system which seems to exist independently of the 
information system being developed to support it, to one 
which seems to have a high degree of overlap, as evidenced 
by the multiple uses of clinical outcome measures for 
informing service planning and delivery. 
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